Add model tests #49

hypercubestart · 2020-10-09T16:54:49Z

memoization PR moved to: #59

gussmith23 · 2020-10-09T18:07:34Z

@hypercubestart I should have warned you! I honestly didn't think about it until now. A little speedup on these tests is actually to be expected, though the speedup we're seeing is perhaps more than I would have guessed. However, the speedups on interpreting larger workloads (i.e. a whole model) should be significant.

gussmith23 · 2020-10-09T18:08:45Z

The reason is because memoization really only helps when expressions are being interpreted multiple times. In simple tests like these, that's not happening, and so memoization is just added overhead. However, in large expressions with repeated subexpressions, we should see big savings (from exponential time to linear).

gussmith23 · 2020-10-09T18:10:46Z

What we need to do is to enable some larger interpreter tests, too. I can write some for you, or you can take a crack at it. Here are my suggestions:

Biggest things we could test right now: Resnet and Mobilenet
We could also write tests to just interpret the first few layers of Resnet/Mobilenet

gussmith23

This looks good so far, just a few comments. We will merge it with the PR that improves our usage of ndarray like we discussed!

src/language/interpreter.rs

gussmith23 · 2020-10-13T16:58:27Z

src/language/from_relay/mod.rs

@@ -1808,4 +1813,126 @@ def @main(%x: Tensor[(3), float32]) -> Tensor[(3), float32] {
 (compute softmax (access (compute softmax (access (access-tensor x) 0)) 0))
 "#
    );
+
+    #[bench]
+    fn mobilenet(b: &mut Bencher) {


Suggested change

fn mobilenet(b: &mut Bencher) {

fn mobilenet_shallow(b: &mut Bencher) {

src/language/from_relay/mod.rs

gussmith23 · 2020-10-14T23:14:32Z

src/language/from_relay/mod.rs

+                    Uniform::new(0f32, 255f32),
+                    &mut tensor_rng,
+                )
+            } else {
+                ndarray::ArrayD::<f32>::random_using(
+                    shape.clone(),
+                    Uniform::new(-1f32, 1f32),


Just to double check: is it expected that, in a normal run of Mobilenet (e.g. in Relay), the image data would be [0, 255] and the weights would be [-1, 1]? That's surprising to me, though I'm not sure why.

Perhaps this reflects more on my own understanding of neural networks...unless we're using pretrained weights, I'm actually unsure why it matters what range the input activations have. Like, is there some operator that specifically expects a [0,255] range? What effect does changing the range from [-1, 1] to [0, 255] have? Or are we just trying to generate values that mimic real values?

FWIW I'm not actually suggesting any change here, haha. Just wondering out loud mostly.

src/language/interpreter.rs

gussmith23

Why did you rename mobilenet-simplified-for-inference.relay? I think I prefer the original name, as it's more descriptive. Specifically, that's not the mobilenet that comes out of Relay; it's the mobilenet that comes out after you run SimplifyInference. Similarly, Resnet is the same way, as are the shallow versions of the networks you created.

I know it seems pedantic, but I am actually now working with the "real" mobilenet (i.e. mobilenet without the nn.batch_norms simplified out), and so i'm probably going to be adding a mobilenet.relay fairly soon! If you think I should name that file differently, though, definitely willing to hear your argument.

As for the other stuff, there are a few code comments to address. One big thing that I didn't put in the comments: I think we should probably move these model tests to their own file in the tests/ directory. They seem more and more like integration tests (i.e. they are testing multiple parts of Glenside rather heavily.) Perhaps test/ingest-and-interpret-relay-models.rs?

src/language/from_relay/mod.rs

gussmith23 · 2020-10-19T16:48:21Z

src/language/from_relay/mod.rs

+                    // TODO: this was a hack because NAN != NAN
+                    // relay_output.mapv_inplace(|x| if x.is_nan() { 0.0 } else { x });
+                    // interpreter_output.mapv_inplace(|x| if x.is_nan() { 0.0 } else { x });


Should we consider adding a check to assert that there are no NaN values? It seems like there shouldn't be any.

NaN values don't appear in the shallow model testing, but previously it appeared for the full model test (possibly because of weird input data)

i think we can come back to this when I do memo + ndarray since its hard to tell currently whether there are Nan values or not :\

Would something like

interpreter_output.iter().all(|v| !v.is_nan())

work?

gussmith23 · 2020-10-19T16:50:59Z

src/language/from_relay/mod.rs

+    // TODO: TESTS ARE IGNORED BECAUSE INTERPRETER TOO SLOW
+    // ====================================================
+    // Creates a Relay-to-Glenside test over a model
+    // (currently the models we test are convolutional and only image-based)
+    // The test does the following:
+    //  1. Reads the $relay_str_path file and parses as relay module
+    //  2. Converts the relay module to glenside
+    //  3. Generates random input (uniform [0, 255]) and parameters (uniform [-1, 1])
+    //  4. Runs relay module through TVM and benchmarks running glenside expr through interpreter
+    //  5. Compare output of using TVM vs interpreter
+    // $test_name: the name of the created test
+    // $relay_str_path: the path of the file containing the Relay code
+    macro_rules! test_model {


Is this the same as benchmark_model, except that it doesn't benchmark? Do you think we need both?

I think we need both if we want benchmark tests for the shallow models (which runs very fast),
and still have full model testing (which currently takes a few hours)

I'm not sure I understand why you can't use benchmark_model for the full models, though. Is it because of the #[ignore] that you're putting on the test in test_model? If that's the problem, take a look at how egg uses macros to match on #[..] expressions. In that case, you could match on #[..] expressions and attach them to the test. Then you could do

#[ignore = "too slow"] benchmark_model!(...full mobilenet...);

ah thats what i was looking for thanks!

src/language/from_relay/mod.rs

tests/mobilenet-relay-to-glenside.rs

gussmith23

Sorry, forgot to mark "request changes"

tests/ingest-and-interpret-relay-models.rs

gussmith23

Looks great!!

hypercubestart added 3 commits October 9, 2020 11:31

done memo

00b2c9c

not working

eb213ed

fix bug

a1f3116

hypercubestart force-pushed the andrew_branch branch from 6009b56 to a1f3116 Compare October 10, 2020 19:55

hypercubestart added 4 commits October 12, 2020 12:10

fix Nan comparison

2900c6d

fix softmax + shallow mobilenet

152a75f

black_box

b578c1d

fmt

1008069

gussmith23 changed the title ~~[WIP] Memoize interpreter~~ Memoize interpreter Oct 13, 2020

gussmith23 reviewed Oct 14, 2020

View reviewed changes

hypercubestart added 3 commits October 15, 2020 21:10

add reset/mobilenet

3e9ab8a

disable benchmark shallow model tests

8b793c9

revert interpreter

ebf0302

hypercubestart changed the title ~~Memoize interpreter~~ Add model tests Oct 16, 2020

refactor

53423a4

gussmith23 reviewed Oct 19, 2020

View reviewed changes

gussmith23 requested changes Oct 19, 2020

View reviewed changes

fix suggestioins

9b32692

gussmith23 requested changes Oct 20, 2020

View reviewed changes

tests/ingest-and-interpret-relay-models.rs Outdated Show resolved Hide resolved

tests/ingest-and-interpret-relay-models.rs Outdated Show resolved Hide resolved

only use one macro

4066f20

gussmith23 approved these changes Oct 20, 2020

View reviewed changes

gussmith23 merged commit 52e2539 into master Oct 20, 2020

gussmith23 deleted the andrew_branch branch November 20, 2021 00:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add model tests #49

Add model tests #49

hypercubestart commented Oct 9, 2020 •

edited

Loading

gussmith23 commented Oct 9, 2020

gussmith23 commented Oct 9, 2020

gussmith23 commented Oct 9, 2020

gussmith23 left a comment

gussmith23 Oct 13, 2020

gussmith23 Oct 14, 2020

gussmith23 left a comment

gussmith23 Oct 19, 2020

hypercubestart Oct 19, 2020 •

edited

Loading

hypercubestart Oct 19, 2020

gussmith23 Oct 19, 2020

gussmith23 Oct 19, 2020

hypercubestart Oct 19, 2020 •

edited

Loading

gussmith23 Oct 19, 2020

hypercubestart Oct 20, 2020

gussmith23 left a comment

gussmith23 left a comment

	fn mobilenet(b: &mut Bencher) {
	fn mobilenet_shallow(b: &mut Bencher) {

Add model tests #49

Add model tests #49

Conversation

hypercubestart commented Oct 9, 2020 • edited Loading

gussmith23 commented Oct 9, 2020

gussmith23 commented Oct 9, 2020

gussmith23 commented Oct 9, 2020

gussmith23 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gussmith23 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hypercubestart Oct 19, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hypercubestart Oct 19, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gussmith23 left a comment

Choose a reason for hiding this comment

gussmith23 left a comment

Choose a reason for hiding this comment

hypercubestart commented Oct 9, 2020 •

edited

Loading

hypercubestart Oct 19, 2020 •

edited

Loading

hypercubestart Oct 19, 2020 •

edited

Loading