Page 3 of 5

Re: Using a Pi to Log Data in a Magnum Energy System

Posted: Tue Jul 31, 2018 2:29 pm
by CharlesGodwin
Hi Lee

sorry to hear about your fire. Fire's are never nice.

You say you have

MS4448PAE....master,slave,
PT-100 charge controller
BMK
Magweb.

I infer from this you have
TWO MS4448PAE
ONE PT-100
ONE BMK
ONE Magweb

But you do not have an AGS Auto Generator Start device.

This is not a problem, I'm just making sure.

I am currently fixing up my Magnum reader to be more indifferent to the configuration it is working with. I working with Yam in BC who has a very large system. I hope to be finished in august.

My reader will be able to generate either a CSV file or a JSON file (for techies who want to do more). When it's closer to ready I'll post it.

I have no experience with the Magweb

Meanwhile if you're interested let me know your email and I'll send you instructions for building the required Magnum network to Pi interface. It's cheap and, i believe, easy to make. There is also a bundle of test software that's not ready for public use.

The only way I have found to send an email in this forum is to post in the thread and then delete as soon as you are contacted.

PS Yes, I'm much further south, only 45 degrees north.

Re: Using a Pi to Log Data in a Magnum Energy System

Posted: Tue Jul 31, 2018 7:00 pm
by CharlesGodwin
@lee
Do you have a ME-ARC remote control?

Re: Using a Pi to Log Data in a Magnum Energy System

Posted: Fri Aug 03, 2018 4:37 pm
by EdwardHomestead
I have the RTR,

No, I do not have the ARC.

Re: Using a Pi to Log Data in a Magnum Energy System

Posted: Fri Aug 03, 2018 5:04 pm
by Willeert
I have not made much progress towards getting an Arduino to communicate with the Magnum system. Please, when you have time, would you also send me the information on the equipment needed to have the Magnum equipment communicate with a Pi. I bought a bunch of 485 parts in an effort to capture the Magnum packets as a first step but so far have had no success.

Thank you

Will

Willeert@gmail.com

Re: Using a Pi to Log Data in a Magnum Energy System

Posted: Fri Aug 03, 2018 7:51 pm
by CharlesGodwin
I have prepared a short document on how to build an RS485 interface to Magnum energy. The document has not been field tested and my have errors. https://goo.gl/u4vaXZ

Re: Using a Pi to Log Data in a Magnum Energy System

Posted: Sat Aug 04, 2018 2:05 am
by Willeert
Charles - Thank You!

Will

Re: Using a Pi to Log Data in a Magnum Energy System

Posted: Wed Sep 05, 2018 3:04 pm
by sagreen83
Interested in this thread, certainly glad I found it and excited that its still somewhat active.

I am on the front end of wanting to do something very similar. I'm thinking that I want to build a network interface to front end both a Magnum inverter and an Outback charge controller. I have an ARC, AGS, BMK and Magweb in the loop. Want to send the data captured by all of the devices (including the Outback) as well as receive update inputs from an upstream system that I will write to both serve as a web controller (emulate the ARC), as well as a data visualization portal.

Charles, im wondering if you have gotten your code to a point yet where you are ready to share the source?

Thanks in advance,
Scott...

Re: Using a Pi to Log Data in a Magnum Energy System

Posted: Wed Sep 05, 2018 8:20 pm
by CharlesGodwin
Thanks for your interest.r.

I too run a hybrid system with a MorningStar charge controller. Concerning your plans - I have no experience emulating an ARC remote. I'm not sure if you can easily add software to the system as an ARC. But you can try. I just can't add any useful information.

I have a web site providing an information dashboard that is integrated into a custom logger I built. All of this was written for my internal version of this code. I wrote it 5 years ago for my use andhave not rewritten the logger and website to work with the new one I've just developed. It's underway but a long way form being done.As far as the code is concerned it is part of the install image available if you read https://goo.gl/u4vaXZ and install the software.

Read to programmer section too.

Re: Using a Pi to Log Data in a Magnum Energy System

Posted: Sat Sep 08, 2018 5:38 pm
by CharlesGodwin
This has been resolved.

I've hit some snags.

although I have all this code running s expected on my old veteran Pi 2 B, It is not running on Pi 3 B systems. 2 testers reported the problem and I have reproduced it with a clean machine. I don't know why yet.

The symptom is there is no data at the USB port. I must have done something to make it work on the old Pi that has slipped from memory. I'll stat testing.

The problem seems to be OS version related. I cab switch OS between two Pi (a 2 B and a 3 B). The problem is with the OS, not the CPU

Re: Using a Pi to Log Data in a Magnum Energy System

Posted: Tue Sep 11, 2018 1:04 am
by sagreen83
Thank you! Was just in the process of installing on my Raspi 3

Please let me know what you find.

Scott...

Re: Using a Pi to Log Data in a Magnum Energy System

Posted: Tue Sep 11, 2018 2:04 am
by CharlesGodwin
Problems have been resolved. Should be ok to use

Re: Using a Pi to Log Data in a Magnum Energy System

Posted: Tue Sep 11, 2018 11:54 am
by sagreen83
Thanks!

I downloaded last night, wondering if what I have has your fixes?

Scott...

Re: Using a Pi to Log Data in a Magnum Energy System

Posted: Tue Sep 11, 2018 11:59 am
by CharlesGodwin
The changes were posted at 17:02 EDT Monday, Sept 10.

Re: Using a Pi to Log Data in a Magnum Energy System

Posted: Tue Sep 11, 2018 12:18 pm
by sagreen83
Thank you, that should do it. I downloaded last night @ 21:06 CST.

Anxious to give this a try. Charles, this is a pretty amazing piece of work you have put together. I really appreciate the amount of effort you have put in to make the install instructions clear! I need to get this on my system to start reading packets, but so far so good.

Seems like there was a ME simulator that was written some time back that reads data packets from storeage, and sends them down the USB channel (on a separate device) so you could do testing without being near the ME inverter. Did you write this simulator?

Scott...

Re: Using a Pi to Log Data in a Magnum Energy System

Posted: Tue Sep 11, 2018 1:12 pm
by CharlesGodwin
I don't know of the emulator as you describe it. But...

I have an ME "emulator" built into my server. When you install the software you are asked to run a test named testrs585. It generates a file with a HEX dump of 100 packets and my server can be started to read that file and serve up that data instead of reading a ME network. The limitation is it is a revolving buffer of data so you get the same packet after the data has been read but it is useful for testing parts of the system and analyzing other users systems. The server defaults to consuming 50 packets for each data request.

The test sample can be increased by running testrs485 with the parameter packets=nnnnn i.e.

Code: Select all

./tests485 packets=1000
To run the server using the generated data use

Code: Select all

./startserver mockdata=true mockdatafile=<fully qualified name of file generated by testrs485>

Re: Using a Pi to Log Data in a Magnum Energy System

Posted: Sat Sep 15, 2018 12:30 am
by sagreen83
I have the server up, and its logging packets. That part is awesome!

The bad part is that I think that only one device on the network can read the packets cause now my normal magweb interface is not showing any statistics for my system.

http://data.magnumenergy.com/MW5278/

I assume that once the server reads the packet the magweb device doesnt get the packets.

Scott...

Re: Using a Pi to Log Data in a Magnum Energy System

Posted: Sat Sep 15, 2018 1:17 am
by sagreen83
I got the magweb working again. Something must have been out of sync in the setup. I unplugged all the connections then got the magweb working again. Next I start the server and the logger and everything was great. Sorry for the false alarm.

Next step: Get the logger running on a device that is offsite and push the JSON to a db and build a web interface!

Will keep you in the loop.

Scott...

Re: Using a Pi to Log Data in a Magnum Energy System

Posted: Sat Sep 15, 2018 3:42 am
by sagreen83
Charles,

I'd like to make some changes to the java for my local setup. Did you happen to use a build tool like Maven or were you running build scripts manually? Would you be willing to share your pom.xml or the shell script you used to build the jar? Would probably save me a lot of time.

Scott...

Re: Using a Pi to Log Data in a Magnum Energy System

Posted: Sat Sep 15, 2018 11:09 am
by CharlesGodwin
sagreen83 wrote:
Sat Sep 15, 2018 1:17 am
I got the magweb working again. Something must have been out of sync in the setup. I unplugged all the connections then got the magweb working again. Next I start the server and the logger and everything was great. Sorry for the false alarm.

Next step: Get the logger running on a device that is offsite and push the JSON to a db and build a web interface!

Will keep you in the loop.

Scott...
@Scott. glad its working. I'd really like a copy of the txt file generated when you run the testrs485 script as I'm trying to fine tune the packet processing and every site seems to have different packet "features" so it's a learning process. Please let me know what equipment you have in the network too.

Thanks
Charles

Re: Using a Pi to Log Data in a Magnum Energy System

Posted: Sat Sep 15, 2018 11:13 am
by CharlesGodwin
sagreen83 wrote:
Sat Sep 15, 2018 3:42 am
Charles,

I'd like to make some changes to the java for my local setup. Did you happen to use a build tool like Maven or were you running build scripts manually? Would you be willing to share your pom.xml or the shell script you used to build the jar? Would probably save me a lot of time.

Scott...
I use the Eclipse IDE http://www.eclipse.org/ on a Win 10 PC and copy the results to a Pi using a Samba mounted drive.

I can't provide the exported project at this time as I have other distractions preventing me from making it a shareable project. However, if you know Eclipse its easy to import the src directory into an Eclipse project. If you don't know Eclipse, then there's a learning curve. What ever you do you will need to include the 4 jar files used for run time. the script mesetup can show you where to find the originals.

Building the jar file is straight forward as there are no special manifest items. It's just a jar file. I use an ant task.

I plan to migrate to Visual Studio Code platform but that's months away.

I'm curious. what do you need to change? I may be able to help or simplify your efforts. I have had a version of this system running for years with a logger that calculates statistics and feeds a website dashboard. Both are primitive but cover the principles.

Re: Using a Pi to Log Data in a Magnum Energy System

Posted: Sat Sep 15, 2018 9:00 pm
by CharlesGodwin
I've just updated the reader using information provided by contributing testers (Thanks, Lee and David) and I think this is pretty close to being version 1.0. But I'd like feedback and generated data from other users. ;)
To update your site download magnum.zip and mesetup to your install directory and run

Code: Select all

bash mesetup

Re: Using a Pi to Log Data in a Magnum Energy System

Posted: Sun Sep 16, 2018 4:26 pm
by sagreen83
We needed to change the date format to ISO 8601 Notation. Was required to get mondodb to import it as a date. We were able to do that and rebuild the .jar file. I have the system now logging locally on the PI and sending the JSON to an offsite mondodb. Starting to code the UI in node.js.

The things that I plan to change are:

1) Get the system to import logs from my Outback FlexMax 80 charge controller. Interested in the process you use to get your charge controllers data into the server.

2) Make the system bi-directional. I really want to be able to change settings on my devices remotely. I know that there are significant changes required to do this.

3) Ultimately I'd like this all to run on an ESP32 (Arduino Variant). I like the raspberry pi for one off projects, but the ESP32 is a bit more imbedded and compact. Has an awesome deep sleep mode that allows you to run the system on a lipo battery.

My configuration is

Magnum MS4448PAE
MagWeb interface
BMK
AGS (w/Honda EU7000is generator)
ME-ARC
Outback FlexMax 80 charge controller
12 VmaxTank 155slr Batteries (420ah)
6 300 watt 72 cell solar panels.

Scott...

p.s. I still owe you that file. I'll get it for you.

Re: Using a Pi to Log Data in a Magnum Energy System

Posted: Sun Sep 16, 2018 8:29 pm
by CharlesGodwin
sagreen83 wrote:
Sun Sep 16, 2018 4:26 pm
Edited for brevity
We needed to change the date format to ISO 8601 Notation. Was required to get mondodb to import it as a date. We were able to do that and rebuild the .jar file. I have the system now logging locally on the PI and sending the JSON to an offsite mondodb. Starting to code the UI in node.js.

I revised the program so the date format is a settable option. Refer to page 6 of my interface document. this will allow others to fine tune to suit their needs too. I for, one am no convinced I need 100th of a second precision on minute by minute logs.

The things that I plan to change are:

1) Get the system to import logs from my Outback FlexMax 80 charge controller. Interested in the process you use to get your charge controllers data into the server.
See below

2) Make the system bi-directional. I really want to be able to change settings on my devices remotely. I know that there are significant changes required to do this.
You're on your own on this one. :D I have enough trouble reading their packets. But, using relay boards, I have a plan to remote start and stop my generator, but that doesn't need Magnum interaction.

3) Ultimately I'd like this all to run on an ESP32 (Arduino Variant). I like the raspberry pi for one off projects, but the ESP32 is a bit more imbedded and compact. Has an awesome deep sleep mode that allows you to run the system on a lipo battery.
remainder deleted....
Plugins
Adding extra "device" data is straight forward but not yet documented. You need to create a java class that implements ca.godwin.magnum.DataReader interface. This class is dynamically loaded at run-time using the plugins= option. Your can have multiple classes by coma delimiting their class names in the option. This class is responsible for returning an ArrayList of objects that implement ca.godwin.magnum.DeviceRecord. How you collect the data in these objects is implementation specific. For example my MorningStar controller has a ModBus interface so I communicate using their published interface, extract the data I want, populate the object so the two methods provide data and I'm done. I also have a class that returns a weather record with current temperature, pressure and humidity. The values are appended after the magnum fields in the order that the plugins are defined and order of each DeviceRecord object in the ArrayList.

NOTE that any extra libraries used by your plugin must be added to the classpath defined in the startserver script. I had to add pi4j directory to support my temperature device and add a .jar for modbus support. I recommend you create your own, separate, script to prevent a future override of the standard script.

Re: Using a Pi to Log Data in a Magnum Energy System

Posted: Tue Sep 18, 2018 12:30 am
by sagreen83
CharlesGodwin wrote:
Sat Sep 15, 2018 11:09 am
@Scott. glad its working. I'd really like a copy of the txt file generated when you run the testrs485 script as I'm trying to fine tune the packet processing and every site seems to have different packet "features" so it's a learning process. Please let me know what equipment you have in the network too.

Thanks
Charles
testRS485-20180917_1920.zip
(1.99 KiB) Downloaded 84 times
@Charles

See attached file created from testrs485

Looks like there are some unknown's in the file. such as:


Bytes read:21:UNKNOWN=>0005140A6419280601EB9B820005000000000000D0
Bytes read:22:UNKNOWN=>400001F8000577000100331E312A73000002025800FE

Again, my configuration is

Magnum MS4448PAE
MagWeb interface
BMK
AGS (w/Honda EU7000is generator)
ME-ARC
Outback FlexMax 80 charge controller
12 VmaxTank 155slr Batteries (420ah)
6 300 watt 72 cell solar panels.

Thanks,
Scott...

Re: Using a Pi to Log Data in a Magnum Energy System

Posted: Tue Sep 18, 2018 12:32 am
by CharlesGodwin
Are you using the latest version. I made some changes in the last 2 days