Move Tensor management to the C layer. #63
Conversation
|
Great work Nick, this is incredible. Definitely wait for someone to review on the C side. Review status: 0 of 17 files reviewed at latest revision, all discussions resolved, some commit checks failed. src/nodejs_kernel_backend.ts, line 133 at r1 (raw file):
should we null the underlying pointer out here? you might run into problems because a lot of our library code calls this src/nodejs_kernel_backend.ts, line 169 at r1 (raw file):
just curious, why is this -1? Comments from Reviewable |
|
Reviewed 17 of 17 files at r1. .vscode/settings.json, line 46 at r1 (raw file):
just curious, what's python in this project that made you add this rule? binding/tfe_utils.h, line 36 at r1 (raw file):
curious: why is this context needed for the data, but not for the shape and type in the methods below? binding/tfe_utils.cc, line 194 at r1 (raw file):
IIUC, this moves data from C to JS. Does this effectively do memcpy ? If it doesn't, maybe you can utilize a similar method for moving data from JS to C as well and avoid memcpy in that case? binding/tfe_utils.cc, line 198 at r1 (raw file):
I wonder if there is perf benefit if you avoid calling src/nodejs_kernel_backend.ts, line 148 at r1 (raw file):
assign the return value of tensorDataSync to Comments from Reviewable |
|
Cool! Nicely done. Just a few small things looking at the C side. |
| void CopyTensorJSBuffer(napi_env env, napi_value wrapped_value, int64_t* shape, | ||
| uint32_t shape_length, TF_DataType dtype, | ||
| napi_value typed_array_value) { | ||
| void CreateTFE_TensorHandleFromTypedArray( |
asimshankar
Apr 3, 2018
Would it make sense for this to return TFE_TensorHandle* instead of being provided one as an output argument?
Would it make sense for this to return TFE_TensorHandle* instead of being provided one as an output argument?
nkreeger
Apr 3, 2018
Author
Member
Yeah makes sense - the napi_api stuff all has outparams. I think I'll enforce TF/TFE specific types should return at the method level. All napi-level stuff should be outparams.
Yeah makes sense - the napi_api stuff all has outparams. I think I'll enforce TF/TFE specific types should return at the method level. All napi-level stuff should be outparams.
| // OpAttr will be used beyond the scope of this function call. Stash ops in a | ||
| // set for re-use instead of dynamically reallocating strings for operations. | ||
| const char* attr_name; | ||
| auto result = ATTR_NAME_SET.find(attr_name_string); |
asimshankar
Apr 3, 2018
If the string copy cost is something of concern (which I'm hoping isn't), notice that the find() call here would copy attr_name_string into an std::string before looking it up.
If the string copy cost is something of concern (which I'm hoping isn't), notice that the find() call here would copy attr_name_string into an std::string before looking it up.
nkreeger
Apr 3, 2018
Author
Member
Thanks - I cleaned this up a little bit.
Thanks - I cleaned this up a little bit.
| if (result == ATTR_NAME_SET.end()) { | ||
| auto insert_result = ATTR_NAME_SET.insert(std::string(attr_name_string)); | ||
| // TODO assert success? | ||
| result = insert_result.first; |
asimshankar
Apr 3, 2018
I think you could simplify these few lines with:
const char* attr_name = ATTR_NAME_SET.insert(attr_name_string).first->c_str();
I think you could simplify these few lines with:
const char* attr_name = ATTR_NAME_SET.insert(attr_name_string).first->c_str();
nkreeger
Apr 3, 2018
Author
Member
Done thank you.
Done thank you.
|
|
||
| TF_AttrType tf_attr_type; | ||
| nstatus = napi_get_value_int32(env, attr_type_value, | ||
| reinterpret_cast<int32_t*>(&tf_attr_type)); |
asimshankar
Apr 3, 2018
Can you use static_cast instead of reinterpret_cast here?
Can you use static_cast instead of reinterpret_cast here?
nkreeger
Apr 3, 2018
Author
Member
Unfortunately, no - the compiler doesn't like going from TF_DataType* to int32_t*.
Unfortunately, no - the compiler doesn't like going from TF_DataType* to int32_t*.
| switch (tf_attr_type) { | ||
| case TF_ATTR_STRING: { | ||
| char value[NAPI_STRING_SIZE]; | ||
| nstatus = napi_get_value_string_utf8(env, type_input_value, value, |
asimshankar
Apr 3, 2018
Strictly speaking string attribute values do not have to be UTF-8 encoded strings, they could be arbitrary byte sequences. In practice this doesn't happen AFAIK, but worth noting in case there is a case where napi_get_value_string_utf8 fails because the value isn't valid UTF-8, but is still a valid TF attribute value.
Strictly speaking string attribute values do not have to be UTF-8 encoded strings, they could be arbitrary byte sequences. In practice this doesn't happen AFAIK, but worth noting in case there is a case where napi_get_value_string_utf8 fails because the value isn't valid UTF-8, but is still a valid TF attribute value.
nkreeger
Apr 3, 2018
Author
Member
OK - I noted in a comment. I think napi_get_value_string_utf8() actually throws an error with a non-utf8 str.
OK - I noted in a comment. I think napi_get_value_string_utf8() actually throws an error with a non-utf8 str.
asimshankar
Apr 4, 2018
Yeah, so the corner case would be that the user will see a UTF-8 encoding error even when the data wasn't meant to be UTF-8 encoded. This is a very rare corner case, so fine to leave as is for now, just giving you a heads up.
Yeah, so the corner case would be that the user will see a UTF-8 encoding error even when the data wasn't meant to be UTF-8 encoded. This is a very rare corner case, so fine to leave as is for now, just giving you a heads up.
| } | ||
|
|
||
| TFE_DeleteTensorHandle(tensor_entry->second); | ||
| tfe_handle_map->erase(tensor_id); |
asimshankar
Apr 3, 2018
Might as well use tensor_entry here?
tfe_handle_map->erase(tensor_entry);
Might as well use tensor_entry here?
tfe_handle_map->erase(tensor_entry);
nkreeger
Apr 3, 2018
Author
Member
Done.
Done.
| ENSURE_NAPI_OK_RETVAL(env, nstatus, nullptr); | ||
|
|
||
| // Push `nullptr` to get a valid pointer in the call to `TFE_Execute()` below. | ||
| std::vector<TFE_TensorHandle*> result_handles; |
asimshankar
Apr 3, 2018
Alternatively:
std::vector<TFE_TensorHandle*> result_handles(num_outputs, nullptr);
And avoid the loop?
Alternatively:
std::vector<TFE_TensorHandle*> result_handles(num_outputs, nullptr);And avoid the loop?
nkreeger
Apr 3, 2018
Author
Member
Done.
Done.
| napi_value shape_value; | ||
| GetTFE_TensorHandleShape(env, handle, &shape_value); | ||
|
|
||
| nstatus = |
asimshankar
Apr 3, 2018
It seems a bit asymmetric that ExecuteOp accepts inputs in terms of integer tensor_ids but provides outputs in the form of objects. I don't feel particularly strongly, but would symmetry be nice?
It seems a bit asymmetric that ExecuteOp accepts inputs in terms of integer tensor_ids but provides outputs in the form of objects. I don't feel particularly strongly, but would symmetry be nice?
nkreeger
Apr 3, 2018
Author
Member
I was thinking the same - however, in our TS layer, we need to know the shape and dtype to create a Tensor in TFJS. Passing all of the information here avoids two traversals from TS->C code to get shape and dtypes from a stored Tensor.
I was thinking the same - however, in our TS layer, we need to know the shape and dtype to create a Tensor in TFJS. Passing all of the information here avoids two traversals from TS->C code to get shape and dtypes from a stored Tensor.
| private: | ||
| TFE_Context* tfe_context; | ||
| std::map<int32_t, TFE_TensorHandle*>* tfe_handle_map; | ||
| int32_t tensor_index; |
asimshankar
Apr 3, 2018
Perhaps call this: next_tensor_id to align with the use of the term "id" elsewhere?
Perhaps call this: next_tensor_id to align with the use of the term "id" elsewhere?
nkreeger
Apr 3, 2018
Author
Member
Done.
Done.
| @@ -246,21 +206,12 @@ void GetTensorData(napi_env env, napi_value context_value, | |||
| } | |||
asimshankar
Apr 3, 2018
When the calls to ENSURE_NAPI_OK above cause an early return, aren't we missing the cleanups (TF_DeleteTensor and maybe TFE_DeleteTensorHandle)?
When the calls to ENSURE_NAPI_OK above cause an early return, aren't we missing the cleanups (TF_DeleteTensor and maybe TFE_DeleteTensorHandle)?
nkreeger
Apr 3, 2018
Author
Member
I've added my TF_AutoTensor class to the TF_Tensor*. I added a TODO to cleanup the TFE_Handle.
I've added my TF_AutoTensor class to the TF_Tensor*. I added a TODO to cleanup the TFE_Handle.
|
@asimshankar Update the C-code PTAL. |
|
LGTM! |
| @@ -258,13 +259,13 @@ void AssignOpAttr(napi_env env, TFE_Op* tfe_op, napi_value attr_value) { | |||
| // OpAttr will be used beyond the scope of this function call. Stash ops in a | |||
| // set for re-use instead of dynamically reallocating strings for operations. | |||
| const char* attr_name; | |||
| auto result = ATTR_NAME_SET.find(attr_name_string); | |||
| std::string attr_name_str(attr_name_string); | |||
| auto result = ATTR_NAME_SET.find(attr_name_str); | |||
asimshankar
Apr 4, 2018
I don't think you need find. Lines 262-267 can be replaced by the one liner:
const char* attr_name = ATTR_NAME_SET.insert(attr_name_string).first->c_str();
Since insert() returns an iterator pointing to the inserted, or pre-existing value.
I don't think you need find. Lines 262-267 can be replaced by the one liner:
const char* attr_name = ATTR_NAME_SET.insert(attr_name_string).first->c_str();Since insert() returns an iterator pointing to the inserted, or pre-existing value.
nkreeger
Apr 4, 2018
Author
Member
Thank you much cleaner.
Thank you much cleaner.
| switch (tf_attr_type) { | ||
| case TF_ATTR_STRING: { | ||
| char value[NAPI_STRING_SIZE]; | ||
| nstatus = napi_get_value_string_utf8(env, type_input_value, value, |
asimshankar
Apr 4, 2018
Yeah, so the corner case would be that the user will see a UTF-8 encoding error even when the data wasn't meant to be UTF-8 encoded. This is a very rare corner case, so fine to leave as is for now, just giving you a heads up.
Yeah, so the corner case would be that the user will see a UTF-8 encoding error even when the data wasn't meant to be UTF-8 encoded. This is a very rare corner case, so fine to leave as is for now, just giving you a heads up.
| tfe_handle_map->insert(pair); | ||
| auto pair = | ||
| std::pair<int32_t, TFE_TensorHandle*>(next_tensor_id_++, tfe_handle); | ||
| tfe_handle_map_.insert(pair); |
asimshankar
Apr 4, 2018
Nit: Make be more idiomatic to do:
tfe_handle_map_.insert(std::make_pair(ned_tensor_id_++, tfe_handle));
Nit: Make be more idiomatic to do:
tfe_handle_map_.insert(std::make_pair(ned_tensor_id_++, tfe_handle));
nkreeger
Apr 4, 2018
Author
Member
Done.
Done.
|
|
||
| tfe_handle_map = new std::map<int32_t, TFE_TensorHandle*>(); | ||
| TFJSBackend::~TFJSBackend() { | ||
| for (auto iter = tfe_handle_map_.begin(); iter != tfe_handle_map_.end(); |
asimshankar
Apr 4, 2018
Could use:
for (auto& kv : tfe_handle_map_) {
TFE_DeleteTensorHandle(kv.second);
}
Could use:
for (auto& kv : tfe_handle_map_) {
TFE_DeleteTensorHandle(kv.second);
}
nkreeger
Apr 4, 2018
Author
Member
Slick - thank you.
Slick - thank you.
| @@ -26,36 +26,43 @@ namespace tfnodejs { | |||
|
|
|||
| class TFJSBackend { | |||
| public: | |||
| TFJSBackend(); | |||
| // TODO(kreeger): Move to a factory and make this private. | |||
| TFJSBackend(napi_env env); | |||
| virtual ~TFJSBackend(); | |||
asimshankar
Apr 4, 2018
If this isn't meant to be sub-classed, I wouldn't bother with a virtual destructor
If this isn't meant to be sub-classed, I wouldn't bother with a virtual destructor
nkreeger
Apr 4, 2018
Author
Member
Thanks updated - scarred from life as an XPCOM developer at Mozilla.
Thanks updated - scarred from life as an XPCOM developer at Mozilla.
|
Review status: 10 of 18 files reviewed at latest revision, 32 unresolved discussions, some commit checks failed. .vscode/settings.json, line 46 at r1 (raw file): Previously, dsmilkov (Daniel Smilkov) wrote…
Nothing Python - reverted this. binding/tfe_utils.h, line 36 at r1 (raw file): Previously, dsmilkov (Daniel Smilkov) wrote…
Context is needed to extract data from the TFE_TensorHandle - shape and dtype are provided w/o needing a Context. binding/tfe_utils.cc, line 194 at r1 (raw file): Previously, dsmilkov (Daniel Smilkov) wrote…
This copies C buffer into a new typed array that is managed by V8. We can circle back and look at buffer management for more tweaks in the future. Right now, all the execution is blocked on allocation of Op execution inside libtensorflow. binding/tfe_utils.cc, line 198 at r1 (raw file): Previously, dsmilkov (Daniel Smilkov) wrote…
PropelML does this. I've added a TODO. src/nodejs_kernel_backend.ts, line 133 at r1 (raw file): Previously, nsthorat (Nikhil Thorat) wrote…
I assumed you meant in src/nodejs_kernel_backend.ts, line 148 at r1 (raw file): Previously, dsmilkov (Daniel Smilkov) wrote…
Done. src/nodejs_kernel_backend.ts, line 169 at r1 (raw file): Previously, nsthorat (Nikhil Thorat) wrote…
Just a random sentinel value. The C code starts at 0. Happy to change to null or something else. Comments from Reviewable |
2101c7b
into
master
Sorry for the large PR, but this one is pretty hard to refactor in small chunks. In a nutshell, this PR moves all Tensor management logic into the C-layer through a new class:
TFJSBackend. Since object wrapping appears expensive, no objects are wrapped. A single static instance ofTFJSBackendis returned when the binding is initialized. All methods exposed on the binding interact with the static reference.Not wrapping objects is a good win for CPU performance on MNIST and a grand slam with Memory (1gb less).
This is the first step of optimizations - almost all CPU blame falls on
TFE_ExecuteOp()according to my profile with this fix.This change is