Export document to jsonFile one by one #131
Conversation
json.go
Outdated
| if err != nil { | ||
| return err | ||
| } | ||
|
|
||
| return ioutil.WriteFile(exportPath, jsonString, os.ModePerm) | ||
| if _, err = f.WriteString("]"); err != nil { |
There was a problem hiding this comment.
When possible, like in this case, I prefer using the := operator
| err = db.ForEach(q, func(doc *d.Document) bool { | ||
| n++ | ||
| jsonByte, err := json.Marshal(doc.AsMap()) | ||
| if err != nil { |
There was a problem hiding this comment.
I think it would be a good idea to recover this error in the outer scope.
If there is any failure inside this function, then the export process would not report an error, but the result would be clearly not correct.
json.go
Outdated
| if n != collectionCount { | ||
| jsonString += "," | ||
| } | ||
| _, err = f.WriteString(jsonString) |
There was a problem hiding this comment.
I guess same thing for this error
There was a problem hiding this comment.
Hey, @letterbeezps, first of all thank you for your contribution, you are doing a great job with the last PRs. The structure looks good, I only left some minor comments.
| } | ||
|
|
||
| jsonString, err := json.Marshal(docs) | ||
| n := 0 | ||
| err = db.ForEach(q, func(doc *d.Document) bool { |
There was a problem hiding this comment.
The Count() and the ForEach() statements run in separate transactions. Thus, if new documents are inserted between them, the ForEach() statement will iterate a number of documents n > collectionCount, which would lead to some problem with the statement:
if n != collectionCount {
jsonString += ","
}
To prevent this issue, we can ensure the number of documents iterated by the ForEach() is exactly collectionCount using Limit():
err = db.ForEach(q.Limit(collectionCount), func(doc *d.Document) bool {
There was a problem hiding this comment.
You remind me that is can also cause problems when documents are deleted between them.
There was a problem hiding this comment.
I have a suggestion for ForEach() function, let consumer() return error instead of bool. If there is any failure inside this function, we can get the error info directly
func (db *DB) ForEach(q *query.Query, consumer func(_ *d.Document) error) error {
q, err := normalizeCriteria(q)
if err != nil {
return err
}
return db.IterateDocs(q, func(doc *d.Document) error {
if err := consumer(doc); err != nil {
return err
}
return nil
})
}
json.go
Outdated
|
|
||
| result, err := db.FindAll(query.NewQuery(collectionName)) | ||
| q := query.NewQuery(collectionName) | ||
| collectionCount, err := db.Count(q) |
There was a problem hiding this comment.
I would just call this variable count, for simplicity
No description provided.