Skip to content

added utf8 support #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from
Closed

added utf8 support #14

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
04ff988
added utf8 support
Buggaboo Sep 4, 2019
9026bc4
I tried explicitly casting with `x as Uint8` and `(Uint) x`
Buggaboo Sep 5, 2019
c77e1b8
Maintain support for 2.4's ffi
Buggaboo Sep 5, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions lib/src/box.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,25 +125,25 @@ class Box<T> {
}

// transform flatbuffers byte array into memory area for C, with a length of a multiple of four
Pointer<Uint8> bufferPtr = allocate(count: ((buffer.length + 3.0) / 4.0).toInt() * 4);
Pointer<Uint8> bufferPtr = Pointer<Uint8>.allocate(count: ((buffer.length + 3.0) / 4.0).toInt() * 4);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm getting compile errors with those:

lib/src/common.dart:8:50: Error: Method not found: 'Pointer.allocate'.
        Pointer<Int32> majorPtr = Pointer<Int32>.allocate(), minorPtr = Pointer<Int32>.allocate(), patchPtr = Pointer<Int32>.allocate();
                                                 ^^^^^^^^

What's your Dart version?

$ dart --version
Dart VM version: 2.4.1

Maybe we have to increase the Dart version requirement?

Copy link
Contributor

@nalenz-objectbox nalenz-objectbox Sep 5, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can confirm @greenrobot's error. If you look at the Dart FFI source file at /usr/lib/dart/lib/ffi/ffi.dart on Linux, you can see that the allocate function is not a static member or constructor of the Pointer class, but just outside of it.

Yet, the FFI SQLite example uses Pointer.allocate as well, despite the underlying source file clearly being different.

Additionally, the master branch of the Dart SDK contains Pointer.allocate in the correct form, see here. This means that either the Dart PPA is deprecated (which is unlikely, as dart --version correctly says 2.4.1 on my machine) or Pointer.allocate is a recent change not yet part of a Dart release, but already in its master branch.

Copy link
Contributor Author

@Buggaboo Buggaboo Sep 5, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

$ dart --version
Dart VM version: 2.5.0-dev.2.1 (Thu Aug 15 16:05:39 2019 +0200) on "macos_x64"

I drink what homebrew offers me.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Uh, so FFI plans to change that!? Is there any way to support both version? If we cannot make this work for both, I think we should stick to the latest release version.

Copy link
Contributor Author

@Buggaboo Buggaboo Sep 5, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reverted the parts that are related to the fromAddress and allocate methods. That specific commit can be (re)reverted once 2.5 is the new stable.

for(int i = 0; i < buffer.length; ++i)
bufferPtr.elementAt(i).store(buffer[i] as int);

// put object into box and free the buffer
checkObx(bindings.obx_box_put(_objectboxBox, propVals[_idPropIdx]["value"], fromAddress(bufferPtr.address), buffer.length, putMode));
checkObx(bindings.obx_box_put(_objectboxBox, propVals[_idPropIdx]["value"], Pointer<Void>.fromAddress(bufferPtr.address), buffer.length, putMode));
bufferPtr.free();
}

getById(int id) {
Pointer<Pointer<Void>> dataPtr = allocate();
Pointer<Int32> sizePtr = allocate();
Pointer<Pointer<Void>> dataPtr = Pointer<Pointer<Void>>.allocate();
Pointer<Int32> sizePtr = Pointer<Int32>.allocate();

// get element with specified id from database
Pointer<Void> txn = bindings.obx_txn_read(_store.ptr);
check(txn != null && txn.address != 0);
checkObx(bindings.obx_box_get(_objectboxBox, id, dataPtr, sizePtr));
checkObx(bindings.obx_txn_close(txn));
Pointer<Uint8> data = fromAddress(dataPtr.load<Pointer<Void>>().address);
Pointer<Uint8> data = Pointer<Uint8>.fromAddress(dataPtr.load<Pointer<Void>>().address);
var size = sizePtr.load<int>();

// transform bytes from memory to Dart byte list
Expand Down
2 changes: 1 addition & 1 deletion lib/src/common.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import "ffi/cstring.dart";

class Common {
static List<int> version() {
Pointer<Int32> majorPtr = allocate(), minorPtr = allocate(), patchPtr = allocate();
Pointer<Int32> majorPtr = Pointer<Int32>.allocate(), minorPtr = Pointer<Int32>.allocate(), patchPtr = Pointer<Int32>.allocate();
bindings.obx_version(majorPtr, minorPtr, patchPtr);
var ret = [majorPtr.load<int>(), minorPtr.load<int>(), patchPtr.load<int>()];
majorPtr.free();
Expand Down
32 changes: 22 additions & 10 deletions lib/src/ffi/cstring.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,24 +1,36 @@
import "dart:ffi";
import "dart:typed_data";
import "package:utf/src/utf8.dart";
import "package:utf/src/utf16.dart";

// TODO check if revamp structs are relevant (https://github.com/dart-lang/sdk/issues/37229)
// wrapper for a null-terminated array of characters in memory ("c-style string")
class CString {
Pointer<Uint8> _ptr;

CString(String dartStr) { // if this constructor is used, ".free" needs to be called on this instance
_ptr = allocate(count: dartStr.length + 1);
for(int i = 0; i < dartStr.length; ++i)
_ptr.elementAt(i).store(dartStr.codeUnitAt(i));
_ptr.elementAt(dartStr.length).store(0);
// if this constructor is used, ".free" needs to be called on this instance
CString(String dartStr) {
List<int> ints = encodeUtf8(dartStr);
var utf8Str = Uint8List.fromList(ints);
_ptr = Pointer<Uint8>.allocate(count: utf8Str.length + 1);
for(int i = 0; i < utf8Str.length; ++i)
_ptr.elementAt(i).store(utf8Str.elementAt(i));
_ptr.elementAt(utf8Str.length).store(0);
}

CString.fromPtr(this._ptr);

String get val {
String ret = "", c;
int i = 0;
while((c = String.fromCharCode(_ptr.elementAt(i++).load<int>())).codeUnitAt(0) != 0) // TODO: unicode support
ret += c;
return ret;
List<int> utf8CodePoints = new List<int>();

int element;

for (int i=0; element != 0; i++) {
element = _ptr.elementAt(i).load<int>();
utf8CodePoints.add(element);
}

return decodeUtf8(utf8CodePoints);
}

String toString() => val;
Expand Down
3 changes: 2 additions & 1 deletion pubspec.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
name: objectbox
version: 0.0.1
version: 0.0.2
description: >-
ObjectBox binding for Dart.
environment:
sdk: '>=2.2.2 <3.0.0'
dependencies:
flat_buffers: ^1.11.0
utf: ^0.9.0
12 changes: 9 additions & 3 deletions test/test.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import "../lib/objectbox.dart";
import "package:utf/src/utf8.dart";

@Entity(id: 1, uid: 1)
class Note {
Expand All @@ -17,11 +18,16 @@ class Note {
main() {
var store = Store([Note]);
var box = Box<Note>(store);
insertNote(box, decodeUtf8([104, 101, 108, 108, 111, 32, 119, 111, 114, 108, 100]));
insertNote(box, decodeUtf8([228, 189, 160, 229, 165, 189, 228, 184, 150, 231, 149, 140, 33]));
insertNote(box, decodeUtf8([65, 104, 111, 106, 32, 115, 118, 196, 155, 116, 101, 33]));
store.close();
}

var note = Note.construct("Hello");
insertNote(Box<Note> box, String str) {
var note = Note.construct(str);
box.put(note);

print("new note got id ${note.id}");
print("refetched note: ${box.getById(note.id)}");

store.close();
}