In recent user group meeting in my city Ahmedabad, I have found that not every user knows difference between these two features of SQL Server. I do not blame any user for not knowing difference between Database Mail and SQLMail as this is very confusing sometime. I will try to explain this concept here.
Database mail is newly introduced concept in SQL Server 2005 and it is replacement of SQLMail of SQL Server earlier version. Database Mail has many enhancement over SQLMail. Database Mail is based on SMTP (Simple Mail Transfer Protocol) and also very fast and reliable where as SQLMail is based on MAPI (Messaging Application Programming Interface). Database mail depends on Service Broker so this service must be enabled for Database Mail. Database Mail can be encrypted for additional security. SQLMail is lesser secure as it can encrypt the message as well anybody can use SMTP to send email. Additionally, for MAPI to be enabled for SQLMail it will require Outlook to be installed. All this leads to potential security threat to database server.
In summary, I suggest if you are using SQLMail, it is right time to upgrade to Database Mail.
If you still want to use SQLMail you will have to enable it with specific commands.
EXEC sp_configure 'SQL Mail XPs', 1
Again, I suggest upgrade to Database Mail and start to stop using SQLMail.