By default, Solaris 11 does not logs any failed login attempt, from any source.
If you want to capture these messages, first of all you need to edit /etc/default/login uncommenting:
#SYSLOG_FAILED_LOGINS=5
and set, if you want, an acceptable threshold (default is 5).
Logging failed attempts from terminal
To log only failed logins from terminal windows:
# touch /var/adm/loginlog # chmod 600 /var/adm/loginlog # chgrp sys /var/adm/loginlog
For the changes to be effective, you need to restart the syslog service:
# svcadm restart system/system-log
Logging failed attempts from any source
To log every failed login:
# touch /var/adm/authlog # chmod 600 /var/adm/authlog # chgrp sys /var/adm/authlog
Edit /etc/syslog.conf:
auth.notice /var/adm/authlog
and refresh the service:
# svcadm refresh system/system-log