Lack of documentation #56
Comments
|
I agree the documentation could be better, though there is a fair amount if you look at the godoc.org rendering of the go documentation. Definitely needs sprucing up. |
|
I think some specific things to do would be:
|
|
Thanks @riannucci, I think I'm a bit lost here: what is the relationship between luci/gae, luci/luci-go and luci/luci-py? If I'm interested in building a web app with a rest api, authentication, using luci/gae data modeling facilities, do I need to use all three luci projects? |
None of this requires or uses code from
Only |
|
@danjacques thanks, when using luci/gae for an app engine classic app, should I simply run the dev server by goapp as usual? The only example I could find is https://github.com/luci/luci-go/tree/master/examples/appengine/helloworld_standard which seems to use |
|
One real perk of The documentation could definitely use some improvement, and it looks like your post inspired riannucci@ to do just that. In the meantime, here is a simple hello world for package init
import (
"fmt"
"net/http"
"strconv"
"github.com/luci/gae/impl/prod"
"github.com/luci/gae/service/datastore"
"golang.org/x/net/context"
)
func init() {
http.HandleFunc("/", hello)
}
type MyThing struct {
ID int64 `gae:"$id"`
Value string
}
func hello(w http.ResponseWriter, r *http.Request) {
c := prod.Use(context.Background(), r)
runHello(c, w)
}
func runHello(c context.Context, w http.ResponseWriter) {
s := &MyThing{Value: "Hello World"}
if err := datastore.Get(c).Put(s); err != nil {
w.WriteHeader(http.StatusInternalServerError)
return
}
w.Header().Add("x-result-id", strconv.FormatInt(s.ID, 10))
fmt.Fprintln(w, "Wrote Hello World struct, got ID %v!", s.ID)
}Here is an example of it being tested: package init
import (
"net/http"
"net/http/httptest"
"testing"
"github.com/luci/gae/impl/memory"
"golang.org/x/net/context"
)
func TestRunHello(t *testing.T) {
c := memory.Use(context.Background())
var rec httptest.ResponseRecorder
runHello(c, &rec)
if rec.Code != http.StatusOK {
t.Fatalf("unexpected code: %d", rec.Code)
}
resultID := rec.Header().Get("x-result-id")
t.Logf("Got ID: %v", resultID)
if resultID == "0" {
t.Fatalf("unexpected zero ID")
}
}Notice the different backends, Hope this helps. |
|
Great, thanks for making things clear. A couple of more questions:
|
That is correct: pRPC is definitely not a REST-ful RPC mechanism. It's an HTTP-centered RPC mechanism meant to parallel the technology behind gRPC, which (as of now) does not work on AppEngine.
Most of the work on |
|
Hi Daniel, I am also just starting to use this package and your response found quite useful. I am mainly going to use luci for in memory testing facility. I have an existing app, and have doubt on the following API use I typically construct the context like so am I supposed to use it like I didn't look further but why need a second context here, if user already passes a valid (at least in production, user should be using a appengine context ) context. thanks, |
|
few more quick qn:
thanks again. |
luci-go uses OpenID Connect protocol, I believe it isn't deprecated. It shouldn't be hard to add Identity Kit integration (by implementing these two interfaces). One issue with using luci-go auth stack, is that it is somewhat more complicated than needed for standalone apps, since it uses LUCI Auth service for user groups, IP whitelists and stuff like that. (It makes sense to use central place for this when you have >2 GAE apps that need them). In particular, a luci-go app would complain that it needs a link to an Auth Service when deployed. It is relatively simple to write a trivial local implementation of authdb.DB interface that doesn't depend on LUCI Auth Service though. If you are still interested in using luci-go in your project, I can add it. |
Awesome! That's a huge reason why the package exists.
So I believe my usage in the example is correct. As you can see in the Use source code, it actually calls As a general rule of thumb, if For example, when using Your namespace setup would look like: import (
"github.com/luci/gae/impl/prod"
"github.com/luci/gae/service/info"
)
// ...
ctx := prod.Use(context.Background(), r)
nctx, err := info.Namespace(ctx, ns)
// use namespaced context for datastore/memcache/.. access
The current API has you supply a import "github.com/luci/gae/service/datastore"
dsInst := datastore.Get(ctx)In this case, all of We actually have an issue open ( #53 ) to un-do this and make it so the
In testing, one sets up preconditions by directly installing them into the testing datastore. You don't need to mock the datastore response because the datastore is actually in the state that you want to begin testing with. In your example, you would set up the users/groups in the test code, then run the function under test and assert the success via datastore queries. Something like (note this is not runnable, just an idea): func TestUpdateGroup(t *testing.T) {
// Set up `impl/memory` testing Context.
// Install precondition Users.
userA := &User{...}
userB := &User{...}
if err := datastore.Get(ctx).Put(userA, userB); err != nil {
t.Fatalf(...)
}
// Install precondition Group.
group := &Group{Users: []*User{userA, userB}}
if err := datastore.Get(ctx).Put(group); err != nil {
t.Fatalf(...)
}
// Run the test.
testUpdateGroups(ctx, ...)
// Assert the group was updated.
datastore.Get(ctx).Get(group)
if group.LastUpdateTime.Before(now) {
t.Fatalf("group didn't update")
}
}In the LUCI project, we use the goconvey test framework, which makes it relatively easy to set up common precondition states in datastore and then perform a bunch of different mutations/tests on those states. Any other test framework will work, though, so pick whatever works for you. Hope this helps! |
|
Hi Daniel, |
|
@vadimsht thanks you for the thorough explanation of auth.
By user groups, do you mean App Engine specific ACL, or general user roles that can be used in any project? If it is the former, then yes, it'd be great if you could add local implementation of authdb.DB so that luci-go auth can be beneficial for outside projects, including ours, too. |
General user roles. They way it work now, one can setup an instance of auth_service (as separate GAE app) and tell a luci-go app to use it (by visiting /admin/settings and following the instructions). Auth Service exposes a UI for group management, like this one: Then luci-go GAE apps can use The problem now, however, is that this link is required for all luci-go apps that use auth system. I was proposing to add a trivial local implementation of authdb.DB (e.g. one that returns false for all Adding a fully functional local version of group management UI is much more time consuming task and not a high priority for me (because we do have auth_service already). |
|
Oh, also the current implementation of the groups imposes some limits. Roughly, the total size of groups database should be <10 MB (because it is kept fully in-memory to make |
|
@vadimsht Wow that's impressive. As you mentioned it has the downside of being dependent on another GAE app. I can see some overlapping in your admin dashboard with Google Endpoints v2, as discussed with @riannucci in a luci-go issue. OAuth and IP-whitelist are covered by Endpoints v2. More stuff like rate control are coming, and the dashboard that you implemented is/will effectively be in Google Cloud Console, where you can manage client ids, etc. @riannucci this is one of the reasons for connecting prpc to Endpoints v2. But we have that REST vs RPC issue. It might be possible to connect prpc to Endpoints v2 without REST though... |
|
@riannucci Is there a supported way of performing data validation when saving entities in luci/gae? I was not able to detect this is luci-go source code (maybe I missed it), and still waiting for the documentation. |
|
Not @riannucci , but the way to validate data is to implement the PropertyLoadSaver interfaces and validate on // Assert that we implement the interface.
var _ datastore.PropertyMap = (*MyType)(nil)
func (t *MyType) Load(pm datastore.PropertyMap) error {
if err := datastore.GetPLS(t).Load(pm); err != nil {
return err
}
return t.validate()
}
func (t *MyType) Save(withMeta bool) (datastore.PropertyMap, error) {
if err := t.validate() err != nil {
return nil, err
}
return datastore.GetPLS(t).Save(withMeta)
} |
|
Thanks @danjacques! A couple of questions:
|
Maybe we're thinking of different types of validation here. As with other datastore implementations, you will want to validate your data as much as possible elsewhere before you tell datastore to store it via The sort of validation that I was referencing in my previous comment is more appropriately employed as protection against schema changes than as a primary means of validating input data.
Not really. Validation seemed custom enough that we left it to the implementer. |
Sorry to be unclear, I was referring to user input validation. Some data access layers have this integrated, I was wondering if this is the case in lucy/gae. |
|
Ah I see - so confirming: there is no specific user input validation provision built into The example that I gave fits more into the layer of database schema constraint enforcement. It's there, and it works, but it's not as efficient or effective as explicitly validating the data before engaging the data layer. |
This is a great repo, this is what GAE Datastore for Go should have been in the first place.
Unfortunately it has no documentation. Yes, I can read the code and see the helpful comments, but there is a lack of documentation for things like how to use the
gaetag.The text was updated successfully, but these errors were encountered: