SVN+SSH On Ubuntu: Locking down security - Passwords

SVN+SSH On Ubuntu: Locking down security - Passwords

Good day,
I have setup a server running Ubuntu 12.04 LTS. I have left the default
SSH port as-is for now, and have set up a DynDNS.org account.
From a machine, running linux, on a separate network, I am able to query
my server via SVN+SSH:
svn --username myName ls
svn+ssh://myName@superdude.dyndns.org/usr/local/svn/repos/private
So far so good: I am queried for the user's password, and an SSH
private/public key pair appears to be generated. However, I am concerned.
I thought the point of private/public keys was that I'd have to provide
the client with the appropriate key in advance. I want to lock down this
server so that the people connecting to my SVN server have both a keyfile
I gave them in advance, and the password for user "myName". Is there
something I'm missing? At this point, all someone appears to need is the
URL to my SVN server, and the password for user "myName", and they are in.
How can I lock this down tighter?
Thank you!