Start a Conversation

This post is more than 5 years old

Solved!

Go to Solution

2677

October 28th, 2011 01:00

XString: problems with nouns? (like ä,ö,ü)

Hello experts,

we succesfully implemented an interface to an EMC Centera system via XAM. We use the .NET Wrapper Version (1.0.57) and our cluster has CentraStar 4.2.0-3881-0-0 installed. So far so good.

I tested different documents, everything worked fine. But then, depending on the document type I wanted to store on the system, I received an exception. Well not really, becaus the errorToken of the exception was always NULL, so I had no clue what was going wrong. The error only occured when loading a file, it seemed that I always could save the file and I also received a valid XUID (see attachment "save.png").

After some "try and error" I finally found (a bug?): I'm saving various metadata in my xSet, one is the title of the document:


xSet.CreateField(FIELDNAME_TITLE, rowDOK.Titel);

I found out, as soon as my document title contains nouns like ä,ö,ü the system fails to load the file (is it a serializing / encoding) problem? Could you please verify my problem and tell me, if I have to encode or do "something special" myself or if it's a problem in the API/VIM?

Any help would be appreciated,

best regards

Ivan Schnyder

3 Attachments

November 27th, 2011 23:00

OK, I see.

Anyway I tried different aproaches, so far what has worked:

Using bytearray with  CreateXStream worked perfectly! I was able to read and write back unicode text as bytes array without any issues. This does mean I have to revert to using only XStream instead of XString

Encoding string with base64 and using this for XString values. Worked perfectly here also (except when encoding Unicode UTF16 string it causes the actual text to be 2 times larger than usual, This can be countered with applying GZipStream afterwards, but I really don't want to go that far, just to be able to store a simple string )

November 24th, 2011 02:00

Hi, I have been checking this out, old post, I know, but it seems there is a problem here.

In XAM documentation it is specified that field name should be of type:

xam_string application/vnd.snia.xam.string Shall be a UTF-8 encoded Unicode

string, with a maximum length of 512 bytes

Each field has a name that is scoped to the XSet. Field names shall be UTF-8 encoded strings (see

[RFC3629]) with a maximum length of 512 bytes. Field names shall be case sensitive, so

.xam.org.snia.field

is distinct from .xam.org.snia.Field. The field names shall not have embedded NULL

characters. To maximize application interoperability, applications are encouraged to use the IDN profile of

stringprep

as defined in [RFC 3491] and [RFC 3454].

Now I made a simple test case with .NET XAM wrapper to see if this is the case:

[

Test]

     

public virtual void TestUnicodeTextFieldRead()

{var wrapper = new XAMWrapper(GetConnectionString());

Trace.WriteLine("Connect with =" + GetConnectionString());

//CONNECT

wrapper.Connect();

          

// WRITE

//now we can write values - get our writer

XAMWriter writer = wrapper.WriteXSet();

//values to test

     

String stringValue1 = "normal text";

String fieldName = "öõüpäöõpoõoõzž";

writer.WriteFieldValue(fieldName, stringValue1); <-- Error here "xam/non-UTF8 parameter"

 

          

//commit changes

writer.Commit();

               

//READ

//XUID now exists

Assert.IsNotNull(writer.XUID);

//get current field value - should be equal to our initial value

          

String field1 = writer.GetFieldValue(fieldName);

 

               

//delete the xset

wrapper.DeleteWriter(writer);Trace.WriteLine("Deleted OK, XUID=" + writer.XUID);

wrapper.Dispose();

          

Trace.WriteLine("Disposed = OK");

          

//check the results

Assert.AreEqual(stringValue1, field1);

}

P.S. Your formatting code is really broken.

I am using my own wrappers on top of this, but basically it calls: 

XAM_CreateString

November 24th, 2011 03:00

Thanks Dmitri for your response!

It's still a problem and we haven't found a solution yet. But just to clarify: the problem is not special characters in the fieldname, I get my error when I'm using special characteres in the fieldcontent.

Something like this (adapted to your code):

String stringValue1 = "äöüaaääöö"

String fieldName = "Title";
writer.WriteFieldValue(fieldName, stringValue1);

Does this work for you? I can save such content, but when I want to load:
String field1 = writer.GetFieldValue(fieldName);
I get an exception...

Thanks, best regards
Ivan

November 24th, 2011 04:00

Yes this works for me (at least it doesn't give an error when reading, but there is an error when I am trying to delete the XSet ! -10214),  here is another test case, if fails in another place (the russian text is incorrect after it was read back also, and it throws an exception when I am trying to delete XSet!  Status -10214)
   
   
   
   
   
   
   
   
   
   
   
   
[Test]
         
public virtual void TestUnicodeTextWriteRead()
{
              
var wrapper = new XAMWrapper(GetConnectionString());
Trace.WriteLine("Connect with =" + GetConnectionString());
//CONNECT
wrapper.Connect();
// WRITE
//now we can write values - get our writer
XAMWriter writer = wrapper.WriteXSet();
//values to test
String stringValue1 = "normal text";
String stringValue2 = "hea küll ja teksti on õrn, mahe mägus ka öõüpäöõpoõoõzž";
String stringValue3 = "Один. Уничтожить вражную надежду на побеждение";
 
writer.WriteFieldValue(
"field1", stringValue1);
writer.WriteFieldValue(
"field2", stringValue2);
writer.WriteFieldValue(
"field3", stringValue3);
//commit changes
writer.Commit();
    
//READ
//XUID now exists
Assert.IsNotNull(writer.XUID);
//get current field value - should be equal to our initial value
String field1 = writer.GetFieldValue("field1"); //value is "normal text";
String field2 = writer.GetFieldValue("field2"); //value is "hea küll ja teksti on õrn, mahe mägus ka öõüpäöõpoõoõzž";
    
String field3 = writer.GetFieldValue("field3"); //value is "?????.??.???.?????????.????" (WRONG!?)
 
//delete the xset
wrapper.DeleteWriter(writer); // <-- Exception is thrown here ! (The rights are available, other test cases work, only this one fails)
Trace.WriteLine("Deleted OK, XUID=" + writer.XUID);
wrapper.Dispose();
              
Trace.WriteLine("Disposed = OK");
 
//check the results
Assert.AreEqual(stringValue1, field1);
Assert.AreEqual(stringValue2, field2);
Assert.AreEqual(stringValue3, field3);
}
I tried different things, here is the actual method:
DllImport("xam.dll")]
public static extern int XAM_CreateString (XAMHandle inHandle, string inName, [MarshalAs(UnmanagedType.U1)] bool inBinding, string inValue);
I can change it to:
[DllImport("xam.dll")]
public static extern int XAM_CreateString (XAMHandle inHandle, byte[] inName, [MarshalAs(UnmanagedType.U1)] bool inBinding, byte[] inValue);
And when passing String values I can use UTF8 encoding to bytes...
But it seems it doesn't work, I get all kind of errors with this.
I also tried also using
[DllImport("xam.dll", CharSet = CharSet.Unicode )]
public static extern int XAM_CreateString (XAMHandle inHandle, string inName, [MarshalAs(UnmanagedType.U1)] bool inBinding, string inValue);
But the results I got with this were even more inconsistent, even the normal text returned in chineese (japaneese maybe) after it was stored in english!
So I really don't know what to do with this right now, if using Unicode UTF8 is not possible, then I have to use alternative approach and encode text with Base64 , can't see any other way right now. Any help anyone ?

Message was edited by: DmitriKuznetsoiv

November 24th, 2011 05:00

Could you send me a set of your versions? I'm working with (64bit):

Centera VIM: 1.0.57

XAMSDK.dll: 1.0.0.0

FPCore.dll: 4.0.660.0
fpos64.dll: 3.2.281.0
fpparser64.dll: 3.2.48.0
FPStreams64:4.0.660.0
FPUtils64.dll: 4.0.660.0
FPXML64.dll: 4.0.660.0

Thanks

November 24th, 2011 05:00

ivan.schnyder wrote:

Could you send me a set of your versions? I'm working with (64bit):

Centera VIM: 1.0.57

XAMSDK.dll: 1.0.0.0

FPCore.dll: 4.0.660.0
fpos64.dll: 3.2.281.0
fpparser64.dll: 3.2.48.0
FPStreams64:4.0.660.0
FPUtils64.dll: 4.0.660.0
FPXML64.dll: 4.0.660.0

Thanks

My versions are:

Centera VIM: 1.0.103.0

XAMSDK.dll: 1.0.0.0

FPCore.dll: 4.0.698.0
fpos64.dll: 3.2.281.0
fpparser.dll: 3.2.48.0
FPStreams:4.0.698.0
FPUtils.dll: 4.0.698.0
FPXML.dll: 4.0.698.0

I think I am using 32 bit versions here...

November 24th, 2011 06:00

Thanks,
so I updated all my dll to the latest (EMC Centera SDK for XAM 1.0 Patch 3 for Windows (32-bit and 64-bit and EMC Centera SDK for XAM VIM 1.0 Patch) - my VIM Version is now 1.0.115 but still the same error...

November 28th, 2011 01:00

Thank you so much Dmitri!

Your hints took me to the solution: I tried the workaround with the XStream. It didn't work till I set the encoding on the array to UTF32. So after it finally worked with the XStream, I wrote myself two little methods and I can know succesfully write / read a normal  "field" with the following code:

// Write

xSet.CreateField(FIELDNAME_TITLE, EncodeToBase64("äöüéàa"))


// Read

xSet.GetField(FIELDNAME_FILETYPE, xStringFieldContent);
string strField = DecodeFromBase64(xStringFieldContent);

// Methods

private string EncodeToBase64(string strToEncode)

{

byte[] toEncodeAsBytes = UTF32Encoding.Default.GetBytes(strToEncode);

return Convert.ToBase64String(toEncodeAsBytes);

}

private string DecodeFromBase64(string strEncodedData)

{

byte[] encodedDataAsBytes = Convert.FromBase64String(strEncodedData);

return UTF32Encoding.Default.GetString(encodedDataAsBytes);

}

Best regards and thank you one more time to analyse my problem!
Ivan

November 28th, 2011 02:00

ivan.schnyder wrote:

Thank you so much Dmitri!

Your hints took me to the solution: I tried the workaround with the XStream. It didn't work till I set the encoding on the array to UTF32. So after it finally worked with the XStream, I wrote myself two little methods and I can know succesfully write / read a normal  "field" with the following code:

// Write

xSet.CreateField(FIELDNAME_TITLE, EncodeToBase64("äöüéàa"))


// Read

xSet.GetField(FIELDNAME_FILETYPE, xStringFieldContent);
string strField = DecodeFromBase64(xStringFieldContent);

// Methods

private string EncodeToBase64(string strToEncode)

{

byte[] toEncodeAsBytes = UTF32Encoding.Default.GetBytes(strToEncode);

return Convert.ToBase64String(toEncodeAsBytes);

}

private string DecodeFromBase64(string strEncodedData)

{

byte[] encodedDataAsBytes = Convert.FromBase64String(strEncodedData);

return UTF32Encoding.Default.GetString(encodedDataAsBytes);

}

Best regards and thank you one more time to analyse my problem!
Ivan

Sure, that's good that it worked for you.

I see  in this example you are using UTF32, but why ? AFAIK C# strings are UTF16, UTF32 doesn't add anything and just wastes space, you can even use UTF8 for this purpose which should work for most possible cases (and usually less space is needed for storage). And I proposed 2 solutions either base64 or XStream, you don't need to use both of them at the same time, just either one of them is enough.

For my solution I went with XStream and UTF16 encoded strings (just for the reason to avoid problems when using XString encoded in Base64 limit of 512 bytes, which means 256 characters for UTF16 and 128 for UTF32 in this case, XStream doesn't have this limit)

November 28th, 2011 02:00

Hi Dmitri

you're absolutly right what concerns strings in c#, they are UTF16 by default, so UTF32 doesn't make any sense at all. But you also wrote:


And I proposed 2 solutions either base64 or XStream, you don't need to use both of them at the same time, just either one of them is enough.


I don't get that one, in my final solution I'm using this method:
xSet.CreateField(FIELDNAME_TITLE,  "äöüéàa") // second parameter is type of XString,
So right know I'm just using the method with XString but before I pass the XString, I encode it to a base64 string, I'm not using the XStream method anymore? Maybe my last answer wasn't clear enough.

Best regards

November 28th, 2011 02:00

Ok, I got a little confused because you mentioned that you were using XStream, but couldn't find it in your example. I guess you are using only XString right now ?

No Events found!

Top