[Question] Is there a plan for a python interface to this?

[thehesiod]

That would be really really cool to speed up performance, and perhaps have a common core across languages.

Reason why I'm so interested is that in our application we need to update millions of files as quickly as possible. I'm currently using aiohttp + botocore v4 auth, and between the overhead of the two I'm limited to ~25 concurrent requests per process and coming from c++ land I know it could do much much more :)

Let me know if I can help!

[JonathanHenson]

Why not use the python sdk to begin with?
On Mar 5, 2016 11:22 AM, "Alexander Mohr" [email protected] wrote:

THAT would be really really cool to speed up performance, and perhaps have
a common core across languages. Let me know if I can help.

—
Reply to this email directly or view it on GitHub
#122.

[thehesiod]

from my experiments the python SDK (botocore/boto3) is too slow and has too much memory overhead. I switched to aiobotocore however its also ~25% slower than going raw aiohttp with botocore auth.

[JonathanHenson]

I suppose the first step is a c-style wrapper. I need to do that anyways. I
can certainly add that to the backlog.
On Mar 5, 2016 12:20 PM, "Alexander Mohr" [email protected] wrote:

from my experiments the python SDK (botocore/boto3) is too slow. There's
too much overhead. I switched to aiobotocore however its also ~25% slower
than going raw aiohttp with botocore auth.

—
Reply to this email directly or view it on GitHub
#122 (comment)
.

[thehesiod]

cool! will be exciting :) could you ping me when that happens? closing this since answered.

[thehesiod]

I think you may not have to write c bindings: https://wiki.python.org/moin/boost.python/SimpleExample. There seem to be various tools available to wrap c++ methods.

[JonathanHenson]

extern "C"

{

extern void cffi_hello(char *name)

{

    return hello(name);

}

}

Thats a c binding.
On Mar 5, 2016 8:01 PM, "Alexander Mohr" [email protected] wrote:

btw, I don't think you need c bindings:
https://gist.github.com/tonyseek/7821993

—
Reply to this email directly or view it on GitHub
#122 (comment)
.

[thehesiod]

sorry, sent wrong example, updated with c++

On Mar 5, 2016, at 8:05 PM, Jonathan M. Henson [email protected] wrote:

extern "C"

{

extern void cffi_hello(char *name)

{

return hello(name);

}

}

Thats a c binding.
On Mar 5, 2016 8:01 PM, "Alexander Mohr" [email protected] wrote:

btw, I don't think you need c bindings:
https://gist.github.com/tonyseek/7821993

—
Reply to this email directly or view it on GitHub
#122 (comment)
.

—
Reply to this email directly or view it on GitHub #122 (comment).

[JonathanHenson]

I like this, boost is a bit of a show stopper for us, but maybe the
dependency is a single header file and in that case, it may work.
On Mar 5, 2016 8:06 PM, "Alexander Mohr" [email protected] wrote:

sorry, sent wrong example, updated with c++

On Mar 5, 2016, at 8:05 PM, Jonathan M. Henson [email protected]
wrote:

extern "C"

{

extern void cffi_hello(char *name)

{

return hello(name);

}

}

Thats a c binding.
On Mar 5, 2016 8:01 PM, "Alexander Mohr" [email protected]
wrote:

btw, I don't think you need c bindings:
https://gist.github.com/tonyseek/7821993

—
Reply to this email directly or view it on GitHub
<
https://github.com/awslabs/aws-sdk-cpp/issues/122#issuecomment-192800681>
.

—
Reply to this email directly or view it on GitHub <
https://github.com/awslabs/aws-sdk-cpp/issues/122#issuecomment-192800870>.

—
Reply to this email directly or view it on GitHub
#122 (comment)
.

[thehesiod]

check this out: https://github.com/pybind/pybind11 https://github.com/pybind/pybind11

found it from this thread: http://stackoverflow.com/questions/13990317/generate-python-bindings-what-methods-programs-to-use http://stackoverflow.com/questions/13990317/generate-python-bindings-what-methods-programs-to-use

btw let me know if I'm pestering too much :)

On Mar 5, 2016, at 8:05 PM, Jonathan M. Henson [email protected] wrote:

extern "C"

{

extern void cffi_hello(char *name)

{

return hello(name);

}

}

Thats a c binding.
On Mar 5, 2016 8:01 PM, "Alexander Mohr" [email protected] wrote:

btw, I don't think you need c bindings:
https://gist.github.com/tonyseek/7821993

—
Reply to this email directly or view it on GitHub
#122 (comment)
.

—
Reply to this email directly or view it on GitHub #122 (comment).

[JonathanHenson]

Thats interesting. I wonder how this works with inheritance and
polymorphism. I need a c binding anyways, but the python port would be
better if it could hook directly in to the OOP code of c++.
On Mar 5, 2016 8:13 PM, "Alexander Mohr" [email protected] wrote:

check this out: https://github.com/pybind/pybind11 <
https://github.com/pybind/pybind11>

found it from this thread:
http://stackoverflow.com/questions/13990317/generate-python-bindings-what-methods-programs-to-use
<
http://stackoverflow.com/questions/13990317/generate-python-bindings-what-methods-programs-to-use

btw let me know if I'm pestering too much :)

On Mar 5, 2016, at 8:05 PM, Jonathan M. Henson [email protected]
wrote:

extern "C"

{

extern void cffi_hello(char *name)

{

return hello(name);

}

}

Thats a c binding.
On Mar 5, 2016 8:01 PM, "Alexander Mohr" [email protected]
wrote:

btw, I don't think you need c bindings:
https://gist.github.com/tonyseek/7821993

—
Reply to this email directly or view it on GitHub
<
https://github.com/awslabs/aws-sdk-cpp/issues/122#issuecomment-192800681>
.

—
Reply to this email directly or view it on GitHub <
https://github.com/awslabs/aws-sdk-cpp/issues/122#issuecomment-192800870>.

—
Reply to this email directly or view it on GitHub
#122 (comment)
.

[thehesiod]

http://pybind11.readthedocs.org/en/latest/classes.html#inheritance http://pybind11.readthedocs.org/en/latest/classes.html#inheritance
http://pybind11.readthedocs.org/en/latest/advanced.html#overriding-virtual-functions-in-python http://pybind11.readthedocs.org/en/latest/advanced.html#overriding-virtual-functions-in-python

pybind/pybind11#105 pybind/pybind11#105
from what I gather casting up is possible, not not down, yet

On Mar 5, 2016, at 8:16 PM, Jonathan M. Henson [email protected] wrote:

Thats interesting. I wonder how this works with inheritance and
polymorphism. I need a c binding anyways, but the python port would be
better if it could hook directly in to the OOP code of c++.
On Mar 5, 2016 8:13 PM, "Alexander Mohr" [email protected] wrote:

check this out: https://github.com/pybind/pybind11 <
https://github.com/pybind/pybind11>

found it from this thread:
http://stackoverflow.com/questions/13990317/generate-python-bindings-what-methods-programs-to-use
<
http://stackoverflow.com/questions/13990317/generate-python-bindings-what-methods-programs-to-use

btw let me know if I'm pestering too much :)

On Mar 5, 2016, at 8:05 PM, Jonathan M. Henson [email protected]
wrote:

extern "C"

{

extern void cffi_hello(char *name)

{

return hello(name);

}

}

Thats a c binding.
On Mar 5, 2016 8:01 PM, "Alexander Mohr" [email protected]
wrote:

btw, I don't think you need c bindings:
https://gist.github.com/tonyseek/7821993

—
Reply to this email directly or view it on GitHub
<
https://github.com/awslabs/aws-sdk-cpp/issues/122#issuecomment-192800681>
.

—
Reply to this email directly or view it on GitHub <
https://github.com/awslabs/aws-sdk-cpp/issues/122#issuecomment-192800870>.

—
Reply to this email directly or view it on GitHub
#122 (comment)
.

—
Reply to this email directly or view it on GitHub #122 (comment).

[prestomation]

Hi @thehesiod,
Have you benchmarked this C++ SDK with the same test cases that you used to determine that botocore is too slow for you?

We would love to see your test cases and performance metrics to understand that this SDK with python bindings would actually be faster for you.

[JonathanHenson]

I think that's a great idea. Make sure you trst using async methods so io
latency doesn't skew the results as much.
On Mar 5, 2016 8:54 PM, "Preston Tamkin" [email protected] wrote:

Hi @thehesiod https://github.com/thehesiod,
Have you benchmarked this C++ SDK with the same test cases that you used
to determine that botocore is too slow for you?

We would love to see your test cases and performance metrics to understand
that this SDK would actually be faster for you.

—
Reply to this email directly or view it on GitHub
#122 (comment)
.

[thehesiod]

@prestomation I have not yet, I was assuming the c++ libs would be faster :) Basically my test was going to be writing a simple wrapper around the c++ lib to do the test. If I get around to it I'll report back. thanks for the time guys!