What's the point of having pointers in Go?

I know that pointers in Go allow mutation of a function's arguments, but wouldn't it have been simpler if they adopted just references (with appropriate const or mutable qualifiers). Now we have pointers and for some built-in types like maps and channels implicit pass by reference.

Am I missing something or are pointers in Go just an unnecessary complication?