★ wanayoo — archive 1999 https://github.com/tensorflow/tfjs-node/pull/63Nouvelle recherche | Portail wanayoo
Skip to content
This repository has been archived by the owner. It is now read-only.

Move Tensor management to the C layer. #63

Merged
merged 21 commits into from Apr 4, 2018
Merged

Move Tensor management to the C layer. #63

merged 21 commits into from Apr 4, 2018

Conversation

@nkreeger
Copy link
Member

@nkreeger nkreeger commented Apr 2, 2018 •

Sorry for the large PR, but this one is pretty hard to refactor in small chunks. In a nutshell, this PR moves all Tensor management logic into the C-layer through a new class: TFJSBackend. Since object wrapping appears expensive, no objects are wrapped. A single static instance of TFJSBackend is returned when the binding is initialized. All methods exposed on the binding interact with the static reference.

Not wrapping objects is a good win for CPU performance on MNIST and a grand slam with Memory (1gb less).

This is the first step of optimizations - almost all CPU blame falls on TFE_ExecuteOp() according to my profile with this fix.


This change is Reviewable

nkreeger added 17 commits Apr 1, 2018
wip
wip
wip
wip
wip
@nsthorat
Copy link
Collaborator

@nsthorat nsthorat commented Apr 3, 2018

:lgtm_strong:

Great work Nick, this is incredible. Definitely wait for someone to review on the C side.


Review status: 0 of 17 files reviewed at latest revision, all discussions resolved, some commit checks failed.


src/nodejs_kernel_backend.ts, line 133 at r1 (raw file):

  dispose(): void {
    throw new Error('Method not implemented.');

should we null the underlying pointer out here? you might run into problems because a lot of our library code calls this


src/nodejs_kernel_backend.ts, line 169 at r1 (raw file):

    if (!this.tensorMap.has(dataId)) {
      this.tensorMap.set(
          dataId, {shape, dtype: this.getTFDType(dtype), values: null, id: -1});

just curious, why is this -1?


Comments from Reviewable

@dsmilkov
Copy link
Member

@dsmilkov dsmilkov commented Apr 3, 2018

:lgtm_strong: Really nice Nick!


Reviewed 17 of 17 files at r1.
Review status: all files reviewed at latest revision, 2 unresolved discussions, some commit checks failed.


.vscode/settings.json, line 46 at r1 (raw file):

  "typescript.tsdk": "node_modules/typescript/lib",
  "clang-format.executable": "${workspaceRoot}/node_modules/.bin/clang-format",
  "python.linting.pylintEnabled": false

just curious, what's python in this project that made you add this rule?


binding/tfe_utils.h, line 36 at r1 (raw file):

// TF/TFE pointers.
void CopyTFE_TensorHandleDataToTypedArray(napi_env env,
                                          TFE_Context* tfe_context,

curious: why is this context needed for the data, but not for the shape and type in the methods below?


binding/tfe_utils.cc, line 194 at r1 (raw file):

  napi_value array_buffer_value;
  nstatus = napi_create_external_arraybuffer(env, data, byte_length, nullptr,

IIUC, this moves data from C to JS. Does this effectively do memcpy ? If it doesn't, maybe you can utilize a similar method for moving data from JS to C as well and avoid memcpy in that case?


binding/tfe_utils.cc, line 198 at r1 (raw file):

  ENSURE_NAPI_OK(env, nstatus);

  nstatus = napi_create_typedarray(env, array_type, length, array_buffer_value,

I wonder if there is perf benefit if you avoid calling create_typedarray and return an arraybuffer directly? Then in js land, you can simply wrap the buffer with a typedArray (e.g. new Float32Array(arrayBuffer))


src/nodejs_kernel_backend.ts, line 148 at r1 (raw file):

      return info.values;
    } else {
      return this.binding.tensorDataSync(info.id);

assign the return value of tensorDataSync to info.values, so if someone calls readSync() twice, the second time is cached.


Comments from Reviewable

Copy link

@asimshankar asimshankar left a comment

Cool! Nicely done. Just a few small things looking at the C side.

void CopyTensorJSBuffer(napi_env env, napi_value wrapped_value, int64_t* shape,
uint32_t shape_length, TF_DataType dtype,
napi_value typed_array_value) {
void CreateTFE_TensorHandleFromTypedArray(

This comment has been minimized.

@asimshankar

asimshankar Apr 3, 2018

Would it make sense for this to return TFE_TensorHandle* instead of being provided one as an output argument?

This comment has been minimized.

@nkreeger

nkreeger Apr 3, 2018
Author Member

Yeah makes sense - the napi_api stuff all has outparams. I think I'll enforce TF/TFE specific types should return at the method level. All napi-level stuff should be outparams.

// OpAttr will be used beyond the scope of this function call. Stash ops in a
// set for re-use instead of dynamically reallocating strings for operations.
const char* attr_name;
auto result = ATTR_NAME_SET.find(attr_name_string);

This comment has been minimized.

@asimshankar

asimshankar Apr 3, 2018

If the string copy cost is something of concern (which I'm hoping isn't), notice that the find() call here would copy attr_name_string into an std::string before looking it up.

This comment has been minimized.

@nkreeger

nkreeger Apr 3, 2018
Author Member

Thanks - I cleaned this up a little bit.

if (result == ATTR_NAME_SET.end()) {
auto insert_result = ATTR_NAME_SET.insert(std::string(attr_name_string));
// TODO assert success?
result = insert_result.first;

This comment has been minimized.

@asimshankar

asimshankar Apr 3, 2018

I think you could simplify these few lines with:

const char* attr_name = ATTR_NAME_SET.insert(attr_name_string).first->c_str();

This comment has been minimized.

@nkreeger

nkreeger Apr 3, 2018
Author Member

Done thank you.


TF_AttrType tf_attr_type;
nstatus = napi_get_value_int32(env, attr_type_value,
reinterpret_cast<int32_t*>(&tf_attr_type));

This comment has been minimized.

@asimshankar

asimshankar Apr 3, 2018

Can you use static_cast instead of reinterpret_cast here?

This comment has been minimized.

@nkreeger

nkreeger Apr 3, 2018
Author Member

Unfortunately, no - the compiler doesn't like going from TF_DataType* to int32_t*.

switch (tf_attr_type) {
case TF_ATTR_STRING: {
char value[NAPI_STRING_SIZE];
nstatus = napi_get_value_string_utf8(env, type_input_value, value,

This comment has been minimized.

@asimshankar

asimshankar Apr 3, 2018

Strictly speaking string attribute values do not have to be UTF-8 encoded strings, they could be arbitrary byte sequences. In practice this doesn't happen AFAIK, but worth noting in case there is a case where napi_get_value_string_utf8 fails because the value isn't valid UTF-8, but is still a valid TF attribute value.

This comment has been minimized.

@nkreeger

nkreeger Apr 3, 2018
Author Member

OK - I noted in a comment. I think napi_get_value_string_utf8() actually throws an error with a non-utf8 str.

This comment has been minimized.

@asimshankar

asimshankar Apr 4, 2018

Yeah, so the corner case would be that the user will see a UTF-8 encoding error even when the data wasn't meant to be UTF-8 encoded. This is a very rare corner case, so fine to leave as is for now, just giving you a heads up.

}

TFE_DeleteTensorHandle(tensor_entry->second);
tfe_handle_map->erase(tensor_id);

This comment has been minimized.

@asimshankar

asimshankar Apr 3, 2018

Might as well use tensor_entry here?
tfe_handle_map->erase(tensor_entry);

This comment has been minimized.

@nkreeger

nkreeger Apr 3, 2018
Author Member

Done.

ENSURE_NAPI_OK_RETVAL(env, nstatus, nullptr);

// Push `nullptr` to get a valid pointer in the call to `TFE_Execute()` below.
std::vector<TFE_TensorHandle*> result_handles;

This comment has been minimized.

@asimshankar

asimshankar Apr 3, 2018

Alternatively:

std::vector<TFE_TensorHandle*> result_handles(num_outputs, nullptr);

And avoid the loop?

This comment has been minimized.

@nkreeger

nkreeger Apr 3, 2018
Author Member

Done.

napi_value shape_value;
GetTFE_TensorHandleShape(env, handle, &shape_value);

nstatus =

This comment has been minimized.

@asimshankar

asimshankar Apr 3, 2018

It seems a bit asymmetric that ExecuteOp accepts inputs in terms of integer tensor_ids but provides outputs in the form of objects. I don't feel particularly strongly, but would symmetry be nice?

This comment has been minimized.

@nkreeger

nkreeger Apr 3, 2018
Author Member

I was thinking the same - however, in our TS layer, we need to know the shape and dtype to create a Tensor in TFJS. Passing all of the information here avoids two traversals from TS->C code to get shape and dtypes from a stored Tensor.

private:
TFE_Context* tfe_context;
std::map<int32_t, TFE_TensorHandle*>* tfe_handle_map;
int32_t tensor_index;

This comment has been minimized.

@asimshankar

asimshankar Apr 3, 2018

Perhaps call this: next_tensor_id to align with the use of the term "id" elsewhere?

This comment has been minimized.

@nkreeger

nkreeger Apr 3, 2018
Author Member

Done.

@@ -246,21 +206,12 @@ void GetTensorData(napi_env env, napi_value context_value,
}

This comment has been minimized.

@asimshankar

asimshankar Apr 3, 2018

When the calls to ENSURE_NAPI_OK above cause an early return, aren't we missing the cleanups (TF_DeleteTensor and maybe TFE_DeleteTensorHandle)?

This comment has been minimized.

@nkreeger

nkreeger Apr 3, 2018
Author Member

I've added my TF_AutoTensor class to the TF_Tensor*. I added a TODO to cleanup the TFE_Handle.

@nkreeger
Copy link
Member Author

@nkreeger nkreeger commented Apr 3, 2018

@asimshankar Update the C-code PTAL.

Copy link

@asimshankar asimshankar left a comment

LGTM!

@@ -258,13 +259,13 @@ void AssignOpAttr(napi_env env, TFE_Op* tfe_op, napi_value attr_value) {
// OpAttr will be used beyond the scope of this function call. Stash ops in a
// set for re-use instead of dynamically reallocating strings for operations.
const char* attr_name;
auto result = ATTR_NAME_SET.find(attr_name_string);
std::string attr_name_str(attr_name_string);
auto result = ATTR_NAME_SET.find(attr_name_str);

This comment has been minimized.

@asimshankar

asimshankar Apr 4, 2018

I don't think you need find. Lines 262-267 can be replaced by the one liner:

const char* attr_name = ATTR_NAME_SET.insert(attr_name_string).first->c_str();

Since insert() returns an iterator pointing to the inserted, or pre-existing value.

This comment has been minimized.

@nkreeger

nkreeger Apr 4, 2018
Author Member

Thank you much cleaner.

switch (tf_attr_type) {
case TF_ATTR_STRING: {
char value[NAPI_STRING_SIZE];
nstatus = napi_get_value_string_utf8(env, type_input_value, value,

This comment has been minimized.

@asimshankar

asimshankar Apr 4, 2018

Yeah, so the corner case would be that the user will see a UTF-8 encoding error even when the data wasn't meant to be UTF-8 encoded. This is a very rare corner case, so fine to leave as is for now, just giving you a heads up.

tfe_handle_map->insert(pair);
auto pair =
std::pair<int32_t, TFE_TensorHandle*>(next_tensor_id_++, tfe_handle);
tfe_handle_map_.insert(pair);

This comment has been minimized.

@asimshankar

asimshankar Apr 4, 2018

Nit: Make be more idiomatic to do:

tfe_handle_map_.insert(std::make_pair(ned_tensor_id_++, tfe_handle));

This comment has been minimized.

@nkreeger

nkreeger Apr 4, 2018
Author Member

Done.


tfe_handle_map = new std::map<int32_t, TFE_TensorHandle*>();
TFJSBackend::~TFJSBackend() {
for (auto iter = tfe_handle_map_.begin(); iter != tfe_handle_map_.end();

This comment has been minimized.

@asimshankar

asimshankar Apr 4, 2018

Could use:

for (auto& kv : tfe_handle_map_) {
  TFE_DeleteTensorHandle(kv.second);
}

This comment has been minimized.

@nkreeger

nkreeger Apr 4, 2018
Author Member

Slick - thank you.

@@ -26,36 +26,43 @@ namespace tfnodejs {

class TFJSBackend {
public:
TFJSBackend();
// TODO(kreeger): Move to a factory and make this private.
TFJSBackend(napi_env env);
virtual ~TFJSBackend();

This comment has been minimized.

@asimshankar

asimshankar Apr 4, 2018

If this isn't meant to be sub-classed, I wouldn't bother with a virtual destructor

This comment has been minimized.

@nkreeger

nkreeger Apr 4, 2018
Author Member

Thanks updated - scarred from life as an XPCOM developer at Mozilla.

nkreeger added 3 commits Apr 4, 2018
@nkreeger
Copy link
Member Author

@nkreeger nkreeger commented Apr 4, 2018

Review status: 10 of 18 files reviewed at latest revision, 32 unresolved discussions, some commit checks failed.


.vscode/settings.json, line 46 at r1 (raw file):

Previously, dsmilkov (Daniel Smilkov) wrote…

just curious, what's python in this project that made you add this rule?

Nothing Python - reverted this.


binding/tfe_utils.h, line 36 at r1 (raw file):

Previously, dsmilkov (Daniel Smilkov) wrote…

curious: why is this context needed for the data, but not for the shape and type in the methods below?

Context is needed to extract data from the TFE_TensorHandle - shape and dtype are provided w/o needing a Context.


binding/tfe_utils.cc, line 194 at r1 (raw file):

Previously, dsmilkov (Daniel Smilkov) wrote…

IIUC, this moves data from C to JS. Does this effectively do memcpy ? If it doesn't, maybe you can utilize a similar method for moving data from JS to C as well and avoid memcpy in that case?

This copies C buffer into a new typed array that is managed by V8. We can circle back and look at buffer management for more tweaks in the future. Right now, all the execution is blocked on allocation of Op execution inside libtensorflow.


binding/tfe_utils.cc, line 198 at r1 (raw file):

Previously, dsmilkov (Daniel Smilkov) wrote…

I wonder if there is perf benefit if you avoid calling create_typedarray and return an arraybuffer directly? Then in js land, you can simply wrap the buffer with a typedArray (e.g. new Float32Array(arrayBuffer))

PropelML does this. I've added a TODO.


src/nodejs_kernel_backend.ts, line 133 at r1 (raw file):

Previously, nsthorat (Nikhil Thorat) wrote…

should we null the underlying pointer out here? you might run into problems because a lot of our library code calls this

I assumed you meant in disposeData()? I've removed it from the hash.


src/nodejs_kernel_backend.ts, line 148 at r1 (raw file):

Previously, dsmilkov (Daniel Smilkov) wrote…

assign the return value of tensorDataSync to info.values, so if someone calls readSync() twice, the second time is cached.

Done.


src/nodejs_kernel_backend.ts, line 169 at r1 (raw file):

Previously, nsthorat (Nikhil Thorat) wrote…

just curious, why is this -1?

Just a random sentinel value. The C code starts at 0. Happy to change to null or something else.


Comments from Reviewable

@nkreeger nkreeger merged commit 2101c7b into master Apr 4, 2018
2 checks passed
2 checks passed
cla/google All necessary CLAs are signed
code-review/reviewable 2/1 LGTMs
Details
@nkreeger nkreeger deleted the kreeger-refactor branch Apr 4, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
You can’t perform that action at this time.