There is a problem with the setting of environment variables.

Did you start backupserver with sybase user?

Before starting the backup server, switch to sybase user and execute: source $ Sybase/$ Sybase _ ase/server _ name.cfg.